In-place scene labelling and understanding with implicit scene representation

Semantic labelling is highly correlated with geometry and radiance reconstruction, as scene entities with similar shape and appearance are more likely to come from similar classes. Recent implicit neural reconstruction techniques are appealing as they do not require prior training data, but the same fully self-supervised approach is not possible for semantics because labels are human-defined properties.We extend neural radiance fields (NeRF) to jointly encode semantics with appearance and geometry, so that complete and accurate 2D semantic labels can be achieved using a small amount of in-place annotations specific to the scene. The intrinsic multi-view consistency and smoothness of NeRF benefit semantics by enabling sparse labels to efficiently propagate. We show the benefit of this approach when labels are either sparse or very noisy in room-scale scenes. We demonstrate its advantageous properties in various interesting applications such as an efficient scene labelling tool, novel semantic view synthesis, label denoising, super-resolution, label interpolation and multi-view semantic label fusion in visual semantic mapping systems

ElasticFusion: real-time dense SLAM and light source estimation

We present a novel approach to real-time dense visual SLAM. Our system is capable of capturing comprehensive dense globally consistent surfel-based maps of room scale environments and beyond explored using an RGB-D camera in an incremental online fashion, without pose graph optimisation or any post-processing steps. This is accomplished by using dense frame-tomodel camera tracking and windowed surfel-based fusion coupled with frequent model refinement through non-rigid surface deformations. Our approach applies local model-to-model surface loop closure optimisations as often as possible to stay close to the mode of the map distribution, while utilising global loop closure to recover from arbitrary drift and maintain global consistency. In the spirit of improving map quality as well as tracking accuracy and robustness, we furthermore explore a novel approach to real-time discrete light source detection. This technique is capable of detecting numerous light sources in indoor environments in real-time as a user handheld camera explores the scene. Absolutely no prior information about the scene or number of light sources is required. By making a small set of simple assumptions about the appearance properties of the scene our method can incrementally estimate both the quantity and location of multiple light sources in the environment in an online fashion. Our results demonstrate that our technique functions well in many different environments and lighting configurations. We show that this enables (a) more realistic augmented reality (AR) rendering; (b) a richer understanding of the scene beyond pure geometry and; (c) more accurate and robust photometric trackin

Learning meshes for dense visual SLAM

Estimating motion and surrounding geometry of a moving camera remains a challenging inference problem. From an information theoretic point of view, estimates should get better as more information is included, such as is done in dense SLAM, but this is strongly dependent on the validity of the underlying models. In the present paper, we use triangular meshes as both compact and dense geometry representation. To allow for simple and fast usage, we propose a view-based formulation for which we predict the in-plane vertex coordinates directly from images and then employ the remaining vertex depth components as free variables. Flexible and continuous integration of information is achieved through the use of a residual based inference technique. This so-called factor graph encodes all information as mapping from free variables to residuals, the squared sum of which is minimised during inference. We propose the use of different types of learnable residuals, which are trained end-to-end to increase their suitability as information bearing models and to enable accurate and reliable estimation. Detailed evaluation of all components is provided on both synthetic and real data which confirms the practicability of the presented approach

Wind Signatures In The X-Ray Emission-Line Profiles Of The Late-O Supergiant Zeta Orionis

X-ray line-profile analysis has proved to be the most direct diagnostic of the kinematics and spatial distribution of the very hot plasma around O stars. The Doppler-broadened line profiles provide information about the velocity distribution of the hot plasma, while the wavelength-dependent attenuation across a line profile provides information about the absorption to the hot plasma, thus providing a strong constraint on its physical location. In this paper, we apply several analysis techniques to the emission lines in the Chandra High Energy Transmission Grating Spectrometer (HETGS) spectrum of the late-O supergiant zeta Ori (O9.7 Ib), including the fitting of a simple line-profile model. We show that there is distinct evidence for blueshifts and profile asymmetry, as well as broadening in the X-ray emission lines of zeta Ori. These are the observational hallmarks of a wind-shock X-ray source, and the results for zeta Ori are very similar to those for the earlier O star, zeta Pup, which we have previously shown to be well fit by the same wind-shock line-profile model. The more subtle effects on the line-profile morphologies in zeta Ori, as compared to zeta Pup, are consistent with the somewhat lower density wind in this later O supergiant. In both stars, the wind optical depths required to explain the mildly asymmetric X-ray line profiles imply reductions in the effective opacity of nearly an order of magnitude, which may be explained by some combination of mass-loss rate reduction and large-scale clumping, with its associated porosity-based effects on radiation transfer. In the context of the recent reanalysis of the helium-like line intensity ratios in both zeta Ori and zeta Pup, and also in light of recent work questioning the published mass-loss rates in OB stars, these new results indicate that the X-ray emission from zeta Ori can be understood within the framework of the standard wind-shock scenario for hot stars

MINSTED fluorescence localization and nanoscopy

We introduce MINSTED, a fluorophore localization and super-resolution microscopy concept based on stimulated emission depletion (STED) that provides spatial precision and resolution down to the molecular scale. In MINSTED, the intensity minimum of the STED doughnut, and hence the point of minimal STED, serves as a movable reference coordinate for fluorophore localization. As the STED rate, the background and the required number of fluorescence detections are low compared with most other STED microscopy and localization methods, MINSTED entails substantially less fluorophore bleaching. In our implementation, 200–1,000 detections per fluorophore provide a localization precision of 1–3 nm in standard deviation, which in conjunction with independent single fluorophore switching translates to a ~100-fold improvement in far-field microscopy resolution over the diffraction limit. The performance of MINSTED nanoscopy is demonstrated by imaging the distribution of Mic60 proteins in the mitochondrial inner membrane of human cells

A scalable FPGA-based architecture for depth estimation in SLAM

The current state of the art of Simultaneous Localisation and Mapping, or SLAM, on low power embedded systems is about sparse localisation and mapping with low resolution results in the name of efficiency. Meanwhile, research in this field has provided many advances for information rich processing and semantic understanding, combined with high computational requirements for real-time processing. This work provides a solution to bridging this gap, in the form of a scalable SLAM-specific architecture for depth estimation for direct semi-dense SLAM. Targeting an off-the-shelf FPGA-SoC this accelerator architecture achieves a rate of more than 60 mapped frames/sec at a resolution of 640×480 achieving performance on par to a highly-optimised parallel implementation on a high-end desktop CPU with an order of magnitude improved power consumption. Furthermore, the developed architecture is combined with our previous work for the task of tracking, to form the first complete accelerator for semi-dense SLAM on FPGAs, establishing the state of the art in the area of embedded low-power systems

MINSTED nanoscopy enters the Ångström localization range

We report all-optical, room-temperature localization of fluorophores with precision in the Ångström range. These precisions are attained in a STED microscope, by encircling the fluorophore with the low-intensity edge of the STED donut beam, while constantly increasing the absolute donut power. Individual fluorophores bound to a DNA strand are localized with σ = 4.7 Å, corresponding to a fraction of the fluorophore size, with only 2,000 detected photons. MINSTED fluorescence nanoscopy with single-digit nanometer resolution is exemplified by imaging nuclear pore complexes and the distribution of nuclear lamin in mammalian cells labeled by transient DNA hybridization. Since our experiments yield a localization precision σ = 2.3 Å, estimated for 10,000 detected photons, we anticipate that MINSTED will open up entirely new areas of application in the study of macromolecular complexes in cells

A Coordinated X-ray and Optical Campaign on the Nearest Massive Eclipsing Binary, Delta Ori Aa: I. Overview of the X-ray Spectrum

We present an overview of four phase-constrained Chandra HETGS X-ray observations of Delta Ori A. Delta Ori A is actually a triple system which includes the nearest massive eclipsing spectroscopic binary, Delta Ori Aa, the only such object which can be observed with little phase-smearing with the Chandra gratings. Since the fainter star, Delta Ori Aa2, has a much lower X-ray luminosity than the brighter primary, Delta Ori A provides a unique system with which to test the spatial distribution of the X-ray emitting gas around Delta Ori Aa1 via occultation by the photosphere of and wind cavity around the X-ray dark secondary. Here we discuss the X-ray spectrum and X-ray line profiles for the combined observation, having an exposure time of nearly 500 ksec and covering nearly the entire binary orbit. Companion papers discuss the X-ray variability seen in the Chandra spectra, present new space-based photometry and ground-based radial velocities simultaneous with the X-ray data to better constrain the system parameters, and model the effects of X-rays on the optical and UV spectrum. We find that the X-ray emission is dominated by embedded wind shock emission from star Aa1, with little contribution from the tertiary star Ab or the shocked gas produced by the collision of the wind of Aa1 against the surface of Aa2. We find a similar temperature distribution to previous X-ray spectrum analyses. We also show that the line half-widths are about $0.3-0.5\times$ the terminal velocity of the wind of star Aa1. We find a strong anti-correlation between line widths and the line excitation energy, which suggests that longer-wavelength, lower-temperature lines form farther out in the wind. Our analysis also indicates that the ratio of the intensities of the strong and weak lines of \ion{Fe}{17} and \ion{Ne}{10} are inconsistent with model predictions, which may be an effect of resonance scatteringComment: accepted by ApJ; revised according to ApJ proo